Smartphone-based device for point-of-care diagnostics of pulmonary inflammation using convolutional neural networks (CNNs)

In pulmonary inflammation diseases, like COVID-19, lung involvement and inflammation determine the treatment regime. Respiratory inflammation is typically arisen due to the cytokine storm and the leakage of the vessels for immune cells recruitment. Currently, such a situation is detected by the clinical judgment of a specialist or precisely by a chest CT scan. However, the lack of accessibility to the CT machines in many poor medical centers as well as its expensive service, demands more accessible methods for fast and cheap detection of lung inflammation. Here, we have introduced a novel method for tracing the inflammation and lung involvement in patients with pulmonary inflammation, such as COVID-19, by a simple electrolyte detection in their sputum samples. The presence of the electrolyte in the sputum sample results in the fern-like structures after air-drying. These fern patterns are different in the CT positive and negative cases that are detected by an AI application on a smartphone and using a low-cost and portable mini-microscope. Evaluating 160 patient-derived sputum sample images, this method demonstrated an interesting accuracy of 95%, as confirmed by CT-scan results. This finding suggests that the method has the potential to serve as a promising and reliable approach for recognizing lung inflammatory diseases, such as COVID-19.


Mini-microscope system
To capture fern patterns with the desired resolution using a smartphone camera, we employed a smartphonebased microscopy tool measuring 60 × 60 × 60 mm.The microscope structure was meticulously designed using SOLIDWORKS (Dassault Systèmes) and 3D printed from polylactic acid (PLA) using Ultimaker 2+.The imaging system comprises a plano-convex lens with a 5 mm diameter and a 6 mm focal length serving as an objective lens.Additionally, a commercial acrylic condenser lens compatible with the chosen light-emitting diode was incorporated.Illumination was provided by a 3 V and 5 W white LED, powered by a CR2032 battery (Camelion) series with a 1.2 kΩ resistor to achieve optimal light intensity.The entire electronic circuit is integrated into a Printed Circuit Board, with all lenses, including the smartphone lens and the LED, sharing a common optical axis.
The optical system's spacings were meticulously designed to achieve a 40× magnification within a 5 × 5 mm field of view.This design allows the visualization and processing of the entire fern-containing droplet within the field of view.To activate the illumination system and position the fern sample within the field of view zone, the glass slide containing the fern pattern is inserted into the imaging system.The fern sample is optimally focused within the working distance of the optical system, mitigating the need for manual focusing and minimizing perturbations caused by the user.It's worth noting that the auto-focus feature of the smartphone camera compensates for variations in the optical properties of different smartphones.

AI algorithm development
To detect ferning patterns in saliva on a smartphone, we used EfficentNet-B0 architecture.This model showed a top-1 accuracy of 77.3% and a top-5 accuracy of 93.5%, performing across 1000 classes of the ImageNet database 52 .To achieve this efficiency, EfficientNet B0 employs a compound scaling method to simultaneously adjust its depth, width, and resolution.By using this method, images can be classified more accurately with fewer parameters, resulting in more complex representations.EfficientNet-B0 is a mobile-sized architecture having 5.3 million trainable parameters.Such a complex neural network needs a significant number of images to optimize its parameters while training.Due to our insufficient dataset of 650 images, we utilized "transfer learning" to develop the algorithm.In this regard, EfficientNet-B0 was first pre-trained with 14 million images of ImageNet.
ImageNet is an open-source data set containing 14 million classified images from various categories 53 .Then, we retrained and validated the pre-trained model using our dataset of 650 salivary images derived from 70 participants, categorized into ferning and non-ferning groups.The labeled image dataset was split into 80% (520) for training and 20% (130) for validation.Using an 80:20 convention balances providing adequate new sputum images to fine-tune the model for the desired classification task, while retaining sufficient previously "unseen" images to evaluate model performance on distinguishing ferning versus non-ferning patterns and check for overfitting.Other partition ratios (90:10 and 70:30) were tested during development to validate that this 80:20 allocation provided optimal performance.All the input images were resized to 224 × 224 pixels, and the retraining process was done for 80 training steps (epochs) with a learning rate of 0.001.Cross-entropy as a loss function and accuracy was measured to evaluate the learning process of the model.Training and validation accuracy are the percentages of correctly detected images by the model in the training and validation datasets.The variance between training and validation accuracies was calculated to know if the model was overfitting.At the 80th training step (epoch), the validation cross-entropy/loss and accuracy were 0.102and 98.23%, respectively.The weights corresponding to the best performance were saved and used for the model.
The image processing component outside the Convolutional Neural Network (CNN) encompasses a singular step, entailing the resizing of images to dimensions of 224 * 224 pixels.The neural network comprises two principal segments: the first being feature extraction, employing a pretrained EfficientNet, and the second involving the classification aspect, consisting of four dense layers.Within the initial three layers, Rectified Linear Unit (ReLU) activation is applied to mitigate the issue of gradient vanishing, while the output layer employs the softmax activation function to denote the probability of an image belonging to each respective class.For the training procedure, the cross-entropy loss function is selected, assessing the performance of a classification model outputting probability values within the range of 0 to 1, in alignment with the softmax function output.
The optimization algorithm opted for is RMSProp, utilized to minimize the cross-entropy function.The formulation of this optimization process is expounded upon in the subsequent discourse: 1. Rectified Linear Unit (ReLU) Activation Function Formulation:

Softmax Activation Function Formulation:
where z i is the input to the ith unit, and K is the total number of units in the layer.

Cross-Entropy Loss Function Formulation:
where N is the number of classes, y i is the true probability of class i, and y i is the predicted probability of class i.where θ is the parameter at time t, η is the learning rate, g t is the gradient at time t, E g 2 t is the exponentially weighted moving average of squared gradients up to time t, and ǫ is a small constant added for numerical stability.
Here is the neat sketch of the whole algorithm:

Algorithm evaluation
To evaluate the algorithm, receiving operative characteristic (ROC) curve was plotted.The output of the trained convolutional neural network is a continuous number between 0 and 1, representing the probability of the image belonging to a specific group.The ROC curve is plotted based on different threshold values to find the best operating point based on the purpose.The area under the ROC curve (AUC) indicates the proposed method's ability to distinguish between different classes.Additionally, to assess the ability of the retrained model to predict accurately, 160 patient-derived sputum sample images, consisting of both ferning and non-ferning cohorts, were utilized.

Web application development
A web application was developed to analyze the presence of ferning patterns in salivary images.Initially, a drop of the sputum sample is placed at the center of the sample slide.After air-drying, the slide is inserted into the mini-microscope system.Then, the lens provides enough magnification (40X) to observe ferning patterns through any smartphone camera.Subsequently, images are captured and sent to the developed web application, where AI determines the presence of ferning patterns.Also, the designed web application, featuring a convenient and straightforward graphical user interface, enables users to access previous test records to track their disease progression.

Statistical analysis
A 2 × 2 confusion matrix was plotted to determine specificity sensitivity, positive predictive value (PPV), and negative predictive value (NPV) in comparison to the CT-scan results as the golden standard for validating pulmonary inflammation.www.nature.com/scientificreports/ The obtained data in this study was gathered from at least three independent experiments and analyzed using the statistic software GraphPad Prism 8 and expressed as mean ± standard deviation (SD).Statically significant results were reported when the P values were less than 0.05 for a specific experiment.

Ethics and consent to participate
All methods employed in our study were conducted in strict accordance with relevant guidelines and regulations imposed by the University of Tehran.Informed consents were obtained for all participants who were made aware of the planned publication through consent forms, and all the experiments were conducted following pertinent guidelines approved by the University of Tehran.

Results and discussion
The background biology and microscopic translation of the sputum electrolyte During the COVID-19 disease, the immune system signals many white blood cells (WBCs) to travel to the lung environment and combat the viruses.To facilitate this process, the tiny micro-vessels around the respiratory alveolus dilate and become permeable, allowing the immune cells to traverse.The increased permeability of the vessels leads to the filling of air sacs with blood fluid, resulting in acute respiratory distress syndrome (ARDS) and subsequent lung failure in certain areas.Our hypothesis posits that blood serum infiltration into the lung environment will alter the composition and concentration of sputum components, particularly electrolyte salts such as Na and K (Fig. 1).Currently, chest CT scans are conducted to monitor inflammation and respiratory involvement in patients.The images with signs of glass-ground opacification (GGO), patchy consolidation, etc., are fallen into this category (Fig. 2A).
Following the CT results, patients were categorized into CT positive and CT negative groups (Fig. 2A).Subsequently, fasting sputum samples were collected from the patients, and an electrolyte analyzer measured their sodium and potassium ions.Sodium and potassium are primary electrolytes in blood serum.Conditions such as hypernatremia and hyperkalemia can contribute to variations in these electrolytes in the blood, potentially leading to various dangerous diseases.
The concentration of the Na and K ions can be observed in Fig. 2B-D.Notably, a significant increase in the concentration of both sodium and potassium is observable in the sputum samples from CT-positive patients.While the average concentration of Na and K in the CT-negative samples is approximately 29 mmol/L and 26 mmol/L, respectively, these values rise to about 53 mmol/L and 46 mmol/L in CT-positive cases, reflecting an increase of approximately 82% and 77%, respectively.These findings support the hypothesis that the electrolytes released into the respiratory system due to inflammation and subsequent vasodilation can be detected in the sputum samples.www.nature.com/scientificreports/As the next step, the increased salt concentration in the sputum was translated into a graphical picture (Fig. 2E).In this regard, a drop of the sputum sample was air-dried on a glass slide and then imaged by a minimicroscope.As shown in Fig. 2F, the branchy and fern-like patterns due to the crystallization of the Na and K salts could be seen in the dried samples.Moreover, in the sputum samples of the CT-positive patients, the fern patterns occupy more area of the whole drop, which confirms the previously obtained results.This number for the patients with negative CT is ~ 23% and ~ 61% for the positive cases (Fig. 2G).

Design of the mini-microscope
After dropping the sputum sample at the center of the sample slide (Fig. 3A) and allowing it to air-dry, the slide is inserted into the mini-microscope system (Fig. 3B and C) for visualizing the fern patterns.The portable mini-microscope system consists of two main components: a magnifier lens and the lightning board (Fig. 3D).The lens provides a 40× magnification, sufficient to see an air-dried sputum sample with a diameter of 5 mm.The lighting system also consists of an LED attached to an electronic board with a condenser lens for uniform sample illumination.The sample slide is placed between the two lenses in an optimized working distance for having the best focus on the fern structures.A microswitch is also placed for automatic on and off of the system as well as fixing the slide at the location.The cap of the system is designed in a way that could be utilized as the stand for holding the smartphone on top of the mini-microscope (Fig. 3E) and giving the possibility for adjusting the mobile camera with the lens.In the end, the taken image is processed by an AI-based application on the smartphone (Fig. 3F).

CNN for intelligent detection of the ferning patterns
In this study, we employed artificial intelligence (AI), more precisely Convolutional Neural Networks (CNNs), to detect fern patterns in the sputum of an individual on a smartphone.A CNN is a deep learning model commonly used for computer vision tasks such as image classification, object detection, and segmentation 29 .Through a combination of convolutional layers, pooling layers, and fully connected layers, CNNs are designed to learn hierarchical representations of visual data automatically.Feature extraction is performed by convolutional filters across the input image, while downsampling reduces spatial dimensionality.After the features have been extracted, the fully connected layers map them to the output classes.CNNs have achieved state-of-the-art performance in various computer vision tasks and have become a cornerstone of deep learning research 54 .In this work, EfficientNet-B0 was utilized as a mobile-sized architecture pre-trained with the ImageNet dataset.This pre-trained model was retrained and validated by 650 labeled salivary images gathered from 70 participants (Fig. 4A).Input images are resized to 224 × 224 pixels through the model.The network is uniformly scaled in depth, width, and resolution than conventional structures through the EfficientNet.As shown in (Fig. 4B), this structure contains 7 main blocks, each containing a varying number of sub-blocks, which is followed with the added classifier layers for retraining and decide about the ferning patterns (Fig. 4C).For transfer learning, the pre-trained EfficientNet was retrained with a learning rate of 0.0001and for 80 training steps (epochs).After 80 steps of training, the model reached an accuracy of 98.23%, while the crossentropy loss function showed a number of 0.102(Fig.4D and E).The difference between validation and training accuracies showed that no overfitting had occurred.The weights corresponding to this performance were saved and utilized for the final model.Additionally, the receiving operative characteristic (ROC) curve was plotted for different thresholds to evaluate the system's diagnostic ability.The area under the ROC curve (AUC) showed a value of 0.99, which is highly acceptable (Fig. 4F).

System performance on patient-derived salivary samples
To evaluate the system's performance in detecting respiratory inflammation through salivary ferning patterns, we gathered 160 saliva sample images out of 70 participants including healthy people.The samples were divided into two cohorts: inflammatory and non-inflammatory.All the samples were tested using our point-of-care smartphone-based AI method, and CT-scan results validated the presence of pulmonary inflammation.Consequently, the 2 × 2 confusion matrix (Fig. 4G) showed an accuracy of 95%.Meanwhile, the specificity and sensitivity were 95.89% and 94.25%, respectively.Further, for this dataset, the proposed method showed a positive predictive value (PPV) of 96.47% and a negative predictive value (NPV) of 93.33% (Table 1).

Discussion
The 95% accuracy of this smartphone-assisted AI approach for diagnosing lung inflammatory diseases via sputum analysis shows strong potential as an accessible point-of-care screening tool.Specifically, the ability to rapidly stratify COVID 19 patients based on presence of respiratory involvement could empower quicker treatment decisions and monitoring of disease progression.While CT imaging is the current gold standard, requirements www.nature.com/scientificreports/for expensive infrastructure and radiologist availability greatly limit access, whereas this prototype system relies only a miniature microscope and mobile phone.However, several limitations in the current methodology need to be considered.The dataset comprised a relatively small number of retrospective samples restricted mostly to confirmed COVID-19 cases from a single medical center.Expanding this framework to incorporate various respiratory illnesses and patient demographics could strengthen real-world validity and generalizability.Another limitation of the current smartphone-based AI system is potential interference in identifying lung inflammation for patients with a history of tobacco or alcohol use.The byproducts of smoking and drinking can directly impact sputum composition, including inducing fern-like patterns unrelated to inflammation.Thus, for accurate implementation in the clinical setting, adequate protocols would need to be developed regarding timing of sputum analysis relative to the patient's last smoke or

Conclusion
In summary, we have introduced a new technique that can serve as an auxiliary method for the current CT scan technique in detecting pulmonary involvement and lung inflammation in patients with conditions like COVID-19.This method has not been applied to other pulmonary diseases such as asthma, COPD, etc.However, given their similar mechanisms to COVID-19, we anticipate that the system will function similarly for them.
The system comprises a portable mini-microscope with 40X optical zoom capability, a glass slide with a designated area for dropping and drying the sputum sample, and an AI-based application on a smartphone for detecting fern patterns on the sputum sample resulting from the crystallization of electrolytes from the blood serum.
To validate our hypothesis regarding the entry of serum electrolytes into the sputum during the inflammation phase of the disease, we initially measured electrolyte concentration (sodium and potassium) in sputum samples from both CT-positive and CT-negative cases.The results ultimately proved our hypothesis that the electrolyte amount is higher in the sputum of CT-positive patients.After that, the fern structures due to the crystallization of the sputum salts were assessed by the mini-microscope with the assistance of a smartphone with an AI-based application.In this regard, the images are sent to a smartphone application to be analyzed using a CNN to detect fern patterns in sputum as a sign of pulmonary inflammation.EfficientNet-B0 was utilized as a mobile-sized CNN architecture, pre-trained with the ImageNet dataset.Using transfer learning, this pre-trained model was retrained and validated by 650 labeled salivary images gathered from 70 participants.Evaluating on 160 patientderived sputum samples, this method showed a noteworthy accuracy of 95% confirmed by CT-scan results.
Consequently, the study proposes a novel method for detecting inflammation and lung involvement in COVID-19 patients by identifying fern patterns in their sputum samples.An affordable and portable minimicroscope can be used to detect fern patterns in sputum samples using this method, which makes it simple and cost-effective.Using AI to detect the fern patterns, this smartphone application can differentiate between CT positive and negative cases.This method could potentially provide an early indication of lung involvement in COVID-19 patients, allowing for earlier intervention and treatment.We believe this portable chip system could be reliable assistance for doctors, especially in medical centers where CT scan facilities are not provided.

Figure 1 .
Figure 1.During the inflammation phase of COVID-19, numerous immune cells are mobilized into the lung environment through the vasodilation process.The vessels surrounding the alveoli undergo increased permeability, allowing the entry of blood contents into the lung environment.This phenomenon has the potential to alter the concentration of sputum components, including electrolyte salts.

Figure 2 .
Figure 2. (A) CT scan from the two patients with COVID-19 disease and with and without lung inflammation.Signs such as glass ground opacification (GGO) in the CT images imply lung involvement.(B-D) Display the results of the Na, K, and NA + K concentrations in the negative and positive CT cases, respectively.(E) Workflow of the designed method for analyzing the fern structures in the dried sputum samples of the patients with different CT scan results.(F) Different ferning patterns in the air-dried samples of the CT positive and negative cases.(G) The percentage of the fern area in the sputum images of the CT positive and negative cases.

Figure 3 .
Figure 3. (A) Sample slide for dropping the sputum sample (B) Mini-microscope system for visualization and imaging of the air-dried sputum sample.(C) The main device and closed view (C-1) of the mini-microscope.(D) Exploded view of the designed mini-microscope for sputum analysis.(E) Placing the smartphone on the mini-microscope and imaging the sputum sample as well as its analysis by the (F) AI-based application.

Figure 4 .
Figure 4. System performance.(A) Computational flow of data in the utilized neural network, for transfer learning, the pre-trained EfficientNet was retrained by using our dataset of 650 salivary images derived from 70 participants.(B) EfficientNet-B0 structure; this mobile-sized architecture contains 7 main blocks, each containing a varying number of sub-blocks.(C) Classifier layers added for retraining and decide about the ferning patterns.(D,E) training and validation curves for accuracy and cross-entropy of the network; after 80 epochs model achieved a validation accuracy of 98.23% on training set and the validation cross-entropy was 0.18.(F) To evaluate the diagnostic ability of the system, receiving operative characteristic (ROC) curve was plotted for different thresholds.The area under the ROC curve (AUC) showed a value of 0.99.(G) the confusion matrices for the test sets when smartphone-based device analyzed the patient-derived samples.True classes are determined by CT-scan results.

Table 1 .
Diagnostic parameters of the proposed method (%). .Analyzing samples only after sufficient clearance time of tobacco/alcohol traces would help prevent false positives for pulmonary involvement.Alternatively, collecting data across patient groups with varying smoker/ drinker statuses could allow retraining the AI model to account for these factors in its diagnostic algorithm.Implementing such measures to eliminate or adjust for smoking/drinking effects will be important future steps for reducing confounders before widespread adoption.